BigQuery user-scoped OAuth creds - #287
Conversation
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughBigQuery now supports authorized-user OAuth credentials alongside service-account credentials. Engine caches use credential-aware bounded LRU keys. SQL clients detect authentication failures and invalidate affected engines. Tests cover credential handling, cache behavior, and exception classification. ChangesOAuth credentials and engine caching
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant DatasourceConfig
participant BigqueryDialect
participant engine_factory
participant SlayerSQLClient
DatasourceConfig->>BigqueryDialect: provide OAuth credential JSON
BigqueryDialect->>engine_factory: build credential-aware engine
engine_factory-->>SlayerSQLClient: return cached or new engine
SlayerSQLClient->>SlayerSQLClient: execute query
SlayerSQLClient->>engine_factory: invalidate engine on authentication failure
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches 💡 1📝 Generate docstrings 💡
🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (3)
tests/dialects/test_bigquery.py (2)
681-691: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider annotating the fixture for the secret scanner.
Betterleaks flags lines 682-689 as Google Application Default Credentials. The finding is a false positive: the values are placeholders and the dict only matches the authorized-user JSON shape. A suppression comment keeps the secret-scanning signal clean. The repository already applies this pattern in
tests/test_engine_factory.pylines 31-33 with# NOSONAR(S2068).🧹 Proposed annotation
def _oauth_info(**overrides) -> dict: - info = { + info = { # noqa: S106 — test fixture; placeholder OAuth grant, not real credentials "type": "authorized_user", "client_id": "cid.apps.googleusercontent.com", "client_secret": "csecret", "refresh_token": "rtok-alice", "token": "access-token-1", "token_uri": "https://oauth2.googleapis.com/token", }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/dialects/test_bigquery.py` around lines 681 - 691, Add the repository-standard NOSONAR(S2068) suppression annotation to the _oauth_info fixture, covering the placeholder authorized-user credential dictionary flagged by Betterleaks. Keep the fixture values and behavior unchanged, following the existing suppression pattern used elsewhere in the tests.Source: Linters/SAST tools
795-854: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winAdd a fingerprint test for a malformed OAuth payload.
credential_fingerprintruns on every cache-key computation throughslayer/sql/engine_factory.pyline 113-124. It must never raise, otherwise an invalid stored datasource breaks engine lookup instead of producing the clearbuild_engineerror._durable_oauth_materialhandles that with theJSONDecodeErrorfallback atslayer/sql/dialects/bigquery.pylines 114-115, but no test pins it.🧪 Proposed test
def test_credential_fingerprint_tolerates_malformed_oauth_json() -> None: """The fingerprint feeds every cache-key lookup, so a bad grant must produce a digest rather than raise. build_engine reports the error.""" ds = DatasourceConfig( name="bq", type="bigquery", oauth_credentials_json="not json at all", ) assert BigqueryDialect().credential_fingerprint(ds)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/dialects/test_bigquery.py` around lines 795 - 854, Add a test alongside the existing credential_fingerprint tests that constructs a Bigquery DatasourceConfig with malformed oauth_credentials_json and asserts BigqueryDialect().credential_fingerprint returns a non-empty fingerprint without raising. Preserve the test’s focus on tolerant cache-key generation for invalid OAuth payloads.slayer/engine/query_engine.py (1)
94-96: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueUse keyword arguments for cache-key calls.
slayer/engine/query_engine.py#L94-L96: Call_engine_cache_keywithdatasource=andconnection_string=.slayer/sql/engine_factory.py#L237-L237: Call_cache_keywithdatasource=andconnection_string=.slayer/sql/engine_factory.py#L264-L265: Call_cache_keywithdatasource=andconnection_string=.As per coding guidelines, “Use keyword arguments for functions with more than one parameter.”
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/engine/query_engine.py` around lines 94 - 96, Update _engine_cache_key in slayer/engine/query_engine.py:94-96 to pass datasource= and connection_string= as keyword arguments. Update both _cache_key calls in slayer/sql/engine_factory.py:237 and 264-265 the same way, preserving the existing argument values.Source: Coding guidelines
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@docs/configuration/datasources.md`:
- Line 174: Update the oauth_credentials_json description to document that the
connection project may come from the URL host or the grant’s quota_project_id;
state that bigquery://<project>/<dataset> is required only when quota_project_id
is absent, and preserve the existing authorized-user credential details.
In `@slayer/sql/client.py`:
- Around line 544-566: Update authentication-failure cleanup around
_discard_engine_on_auth_failure to be asynchronous, disposing and clearing both
_sync_engine and _async_engine before invalidating the shared engine. Invoke
this cleanup from execute, execute_sync, and get_column_types so every public
execution path discards cached engines after authentication failures. Add
coverage for native-async, synchronous, and column-type failures.
In `@slayer/sql/engine_factory.py`:
- Around line 237-246: Synchronize all accesses to _engine_cache with one shared
lock, including lookup, move_to_end, insertion, invalidate_engine(),
_evict_to_limit(), and reset operations. Update the cache flow around the
visible lookup and _build_engine call so construction may occur outside the
lock, but perform a second locked lookup before inserting to reuse an engine
created concurrently and avoid duplicate pools; return the existing entry when
found, otherwise insert and evict while still holding the lock.
In `@tests/dialects/test_bigquery.py`:
- Around line 736-741: In BigqueryDialect.build_engine, move the optional
sqlalchemy-bigquery imports until after OAuth credential parsing and the
missing-project validation, so build_engine(_oauth_ds(),
connection_string="bigquery://") raises the expected ValueError even when the
optional dependency is unavailable. Preserve the existing import behavior for
valid configurations.
---
Nitpick comments:
In `@slayer/engine/query_engine.py`:
- Around line 94-96: Update _engine_cache_key in
slayer/engine/query_engine.py:94-96 to pass datasource= and connection_string=
as keyword arguments. Update both _cache_key calls in
slayer/sql/engine_factory.py:237 and 264-265 the same way, preserving the
existing argument values.
In `@tests/dialects/test_bigquery.py`:
- Around line 681-691: Add the repository-standard NOSONAR(S2068) suppression
annotation to the _oauth_info fixture, covering the placeholder authorized-user
credential dictionary flagged by Betterleaks. Keep the fixture values and
behavior unchanged, following the existing suppression pattern used elsewhere in
the tests.
- Around line 795-854: Add a test alongside the existing credential_fingerprint
tests that constructs a Bigquery DatasourceConfig with malformed
oauth_credentials_json and asserts BigqueryDialect().credential_fingerprint
returns a non-empty fingerprint without raising. Preserve the test’s focus on
tolerant cache-key generation for invalid OAuth payloads.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: c033876c-ab66-40b6-a659-6a6ec55f7e51
📒 Files selected for processing (16)
docs/configuration/datasources.mdslayer/core/models.pyslayer/engine/cache.pyslayer/engine/query_engine.pyslayer/engine/schema_drift.pyslayer/sql/client.pyslayer/sql/dialects/base.pyslayer/sql/dialects/bigquery.pyslayer/sql/engine_factory.pytests/dialects/test_bigquery.pytests/dialects/test_tsql.pytests/integration/test_in_memory_sqlite.pytests/test_engine_factory.pytests/test_query_cache.pytests/test_sql_client.pytests/test_sql_generator.py
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@tests/test_engine_factory.py`:
- Around line 457-462: Update the thread cleanup assertions in the concurrent
access test to verify every worker in threads is no longer alive after the timed
joins, before asserting errors. Preserve the existing errors assertion and
ensure the test fails when any worker remains running or deadlocked.
- Around line 425-428: Update the cache-convergence test around _dispose_quietly
by patching it before the racing calls, then assert it was called with the built
engine that was not returned to either caller. Preserve the existing assertions
for two builds, shared returned engine, and single cache entry, and clean up the
patch before reset_cache().
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 20a2da85-a3d9-47bb-8cc1-792f37301f9c
📒 Files selected for processing (8)
docs/configuration/datasources.mdslayer/engine/query_engine.pyslayer/sql/client.pyslayer/sql/dialects/bigquery.pyslayer/sql/engine_factory.pytests/dialects/test_bigquery.pytests/test_engine_factory.pytests/test_sql_client.py
🚧 Files skipped from review as they are similar to previous changes (6)
- slayer/sql/client.py
- tests/test_sql_client.py
- docs/configuration/datasources.md
- slayer/sql/dialects/bigquery.py
- slayer/engine/query_engine.py
- slayer/sql/engine_factory.py
6c1368a to
ba344b3
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 3
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@slayer/sql/dialects/bigquery.py`:
- Around line 298-305: Update the ValueError in the project validation branch
after assigning project from url.host or info.get("quota_project_id") to mention
both accepted sources: the connection string and OAuth credentials’
quota_project_id. Preserve the existing “must be given in the connection string”
substring so current tests continue to pass.
In `@slayer/sql/engine_factory.py`:
- Around line 258-278: Update the cache-hit path in the engine factory around
_engine_cache lookup to enforce the current configured cache limit: when the
limit is zero, clear the cache and bypass reuse; when it decreases, trim LRU
entries after the hit. Release _cache_lock before disposing any evicted engines,
while preserving normal cache-hit reuse when entries remain within the limit.
- Around line 257-267: Update the engine creation flow around the cache-key and
_build_engine calls to deep-copy DatasourceConfig before deriving
connection_string, then use that same snapshot for _cache_key() and
_build_engine() so credential changes cannot mismatch the cache fingerprint. Add
a coordinated test covering oauth_credentials_json rotation during creation and
verify the resulting engine is cached under the snapshot’s credentials.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 4571d45c-3662-462f-881d-d9b0e38dd370
📒 Files selected for processing (16)
docs/configuration/datasources.mdslayer/core/models.pyslayer/engine/cache.pyslayer/engine/query_engine.pyslayer/engine/schema_drift.pyslayer/sql/client.pyslayer/sql/dialects/base.pyslayer/sql/dialects/bigquery.pyslayer/sql/engine_factory.pytests/dialects/test_bigquery.pytests/dialects/test_tsql.pytests/integration/test_in_memory_sqlite.pytests/test_engine_factory.pytests/test_query_cache.pytests/test_sql_client.pytests/test_sql_generator.py
🚧 Files skipped from review as they are similar to previous changes (11)
- tests/test_query_cache.py
- tests/dialects/test_tsql.py
- slayer/engine/cache.py
- tests/test_sql_generator.py
- slayer/sql/client.py
- slayer/sql/dialects/base.py
- tests/test_sql_client.py
- slayer/engine/schema_drift.py
- slayer/core/models.py
- slayer/engine/query_engine.py
- tests/integration/test_in_memory_sqlite.py
ba344b3 to
1437afb
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
1437afb to
e181daf
Compare
|
Note GitHub couldn't provide a complete incremental comparison for this pull request, so CodeRabbit is performing a full review instead. This review may take a little longer. |
There was a problem hiding this comment.
Actionable comments posted: 3
🧹 Nitpick comments (4)
tests/test_engine_factory.py (3)
556-566: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueConsider sharing the
_oauth_dshelper.
tests/dialects/test_bigquery.pydefines a helper with the same name and the same body shape. The two copies have already diverged: the BigQuery copy accepts additional keyword arguments such asquota_project_id,token, andexpiry.Move one implementation into a shared test helper or a conftest fixture so both files stay in step.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_engine_factory.py` around lines 556 - 566, The duplicated _oauth_ds helper in tests/test_engine_factory.py and tests/dialects/test_bigquery.py should be consolidated into one shared test helper or conftest fixture. Move the implementation to the shared location, preserve support for the BigQuery helper’s additional keyword arguments such as quota_project_id, token, and expiry, and update both test files to reuse it.
291-337: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winConsider an
autousefixture for cache reset.Every test in this class calls
engine_factory.reset_cache()at the start and at the end. The trailing call does not run when an assertion fails, so a failure leaves module-global cache state for later tests.An
autousefixture that resets before and after each test removes the duplication and makes cleanup unconditional.♻️ Proposed fixture
class TestCacheBounding: """Per-identity keys make cache cardinality track *users*, not datasources, so the cache has to be bounded and evictions must actually release the pooled connections.""" + `@pytest.fixture`(autouse=True) + def _clean_cache(self): + engine_factory.reset_cache() + yield + engine_factory.reset_cache() + `@staticmethod` def _lite(n: int) -> DatasourceConfig:🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_engine_factory.py` around lines 291 - 337, Use an autouse fixture for the test class that calls engine_factory.reset_cache() before each test and guarantees a second reset during teardown. Remove the duplicated start/end reset_cache() calls from test_cache_evicts_least_recently_used_over_limit, test_reuse_refreshes_recency, test_eviction_disposes_the_engine, and test_dispose_failure_does_not_break_caching.
287-289: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valuePrefer the
tmp_pathfixture over fixed/tmpdatabase paths. Four datasource helpers hard-code SQLite file names under/tmp. The names are fixed, so two concurrent runs or two users on the same host collide.get_enginedoes not open a connection in most of these tests, so nothing is written today, but any test that later executes a statement would create a world-readable file at a predictable path. Static analysis flags all four sites (hardcoded-tmp-file, CWE-377).
tests/test_engine_factory.py#L287-L289: taketmp_pathin the calling tests and build thedatabasevalue from it instead off"/tmp/slayer-cache-{n}.db".tests/test_engine_factory.py#L411-L413: replace"/tmp/slayer-invalidate.db"with a path derived fromtmp_path.tests/test_engine_factory.py#L439-L441: replace"/tmp/slayer-reset.db"with a path derived fromtmp_path.tests/test_engine_factory.py#L457-L459: replacef"/tmp/slayer-conc-{n}.db"with a path derived fromtmp_path.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@tests/test_engine_factory.py` around lines 287 - 289, Update the datasource helpers and their calling tests in tests/test_engine_factory.py at lines 287-289, 411-413, 439-441, and 457-459 to use the pytest tmp_path fixture when constructing SQLite database paths. Pass tmp_path into the relevant tests/helpers and derive each database filename from it, replacing all fixed /tmp paths while preserving the existing unique filenames and test behavior.Source: Linters/SAST tools
slayer/sql/engine_factory.py (1)
299-314: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low valueConsider bypassing insertion when the cap is
0.With
SLAYER_MAX_CACHED_ENGINES=0, the second lock block inserts the new engine and_take_evictions_over_limit()immediately pops it. Line 312 then disposes the same engine that Line 314 returns.dispose()swaps in a fresh pool, so the returned engine still works, but the insert/evict/dispose cycle is pure overhead on every call.An early check of the cap before insertion removes that cycle and makes "caching disabled" explicit.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@slayer/sql/engine_factory.py` around lines 299 - 314, Update the cache insertion block around _engine_cache and _take_evictions_over_limit to bypass insertion and eviction when the configured maximum cached-engine cap is 0. Return the newly built engine directly in that case, avoiding disposal of the same engine being returned; preserve the existing concurrent-winner and eviction behavior for positive caps.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@slayer/sql/dialects/bigquery.py`:
- Around line 260-264: Update the sa.create_engine call to pass
connection_string explicitly as the url keyword argument, while leaving
credentials_info and pool_pre_ping unchanged.
In `@slayer/sql/engine_factory.py`:
- Around line 352-354: Update the cache-reset disposal loop around
_dispose_quietly so its reason does not include key[0], which may contain
plaintext credentials. Use the existing non-secret credential fingerprint
portion of each cache key as the disposal identifier, preserving the current
warning behavior and cache-reset context.
In `@tests/dialects/test_bigquery.py`:
- Around line 882-889: Fix Ruff findings in
test_build_engine_oauth_validates_before_importing_optional_driver: combine the
nested patch.dict and pytest.raises context managers into a single with
statement to resolve SIM117, correct the file’s import ordering, and remove the
unused ARG002 suppression directive.
---
Nitpick comments:
In `@slayer/sql/engine_factory.py`:
- Around line 299-314: Update the cache insertion block around _engine_cache and
_take_evictions_over_limit to bypass insertion and eviction when the configured
maximum cached-engine cap is 0. Return the newly built engine directly in that
case, avoiding disposal of the same engine being returned; preserve the existing
concurrent-winner and eviction behavior for positive caps.
In `@tests/test_engine_factory.py`:
- Around line 556-566: The duplicated _oauth_ds helper in
tests/test_engine_factory.py and tests/dialects/test_bigquery.py should be
consolidated into one shared test helper or conftest fixture. Move the
implementation to the shared location, preserve support for the BigQuery
helper’s additional keyword arguments such as quota_project_id, token, and
expiry, and update both test files to reuse it.
- Around line 291-337: Use an autouse fixture for the test class that calls
engine_factory.reset_cache() before each test and guarantees a second reset
during teardown. Remove the duplicated start/end reset_cache() calls from
test_cache_evicts_least_recently_used_over_limit, test_reuse_refreshes_recency,
test_eviction_disposes_the_engine, and
test_dispose_failure_does_not_break_caching.
- Around line 287-289: Update the datasource helpers and their calling tests in
tests/test_engine_factory.py at lines 287-289, 411-413, 439-441, and 457-459 to
use the pytest tmp_path fixture when constructing SQLite database paths. Pass
tmp_path into the relevant tests/helpers and derive each database filename from
it, replacing all fixed /tmp paths while preserving the existing unique
filenames and test behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro
Run ID: 697b2782-f53d-42e3-b7f7-899d2918afc7
📒 Files selected for processing (16)
docs/configuration/datasources.mdslayer/core/models.pyslayer/engine/cache.pyslayer/engine/query_engine.pyslayer/engine/schema_drift.pyslayer/sql/client.pyslayer/sql/dialects/base.pyslayer/sql/dialects/bigquery.pyslayer/sql/engine_factory.pytests/dialects/test_bigquery.pytests/dialects/test_tsql.pytests/integration/test_in_memory_sqlite.pytests/test_engine_factory.pytests/test_query_cache.pytests/test_sql_client.pytests/test_sql_generator.py
🚧 Files skipped from review as they are similar to previous changes (11)
- slayer/core/models.py
- tests/test_sql_generator.py
- slayer/engine/schema_drift.py
- tests/integration/test_in_memory_sqlite.py
- tests/dialects/test_tsql.py
- slayer/engine/cache.py
- slayer/engine/query_engine.py
- tests/test_query_cache.py
- slayer/sql/dialects/base.py
- slayer/sql/client.py
- tests/test_sql_client.py
c32da55 to
ed36272
Compare
|
…scoped creds) DEV-1755: engine cache key becomes a 3-tuple (EngineCacheKey) that folds in a per-datasource credential fingerprint, so two datasources differing only in OAuth grant get distinct engines/clients. Conflict resolutions (branch structure wins; #287 semantics ported): - query_engine.py: kept the branch's _Prepared / _run_data_query / _normalize_stage / refresh internals; ported _sql_client_cache_key to delegate to engine_factory._cache_key (returns EngineCacheKey); updated _sql_clients / _ch_version_cache annotations to EngineCacheKey; dropped now-unused _runtime_fingerprint / SQLGenerator / SLAYER_RESERVED_KEYWORDS imports; discarded main's duplicate _cache init and its _infer_aggregated_ format copy (lives in response_meta.py on the branch). - bigquery.py: alias helpers from slayer.sql.naming; kept main's _digest import (credential_fingerprint uses it). - test_bigquery.py / test_tsql.py: kept branch tests, grafted #287's OAuth / credential_fingerprint tests, dropped dead enriched/enrichment imports, keyed fake clients via _sql_client_cache_key (3-tuple). - test_sql_generator.py: updated the get_column_types cache-key tuple to the 3-tuple form + set credentials_json=None on the mock ds.



Summary by CodeRabbit
New Features
Documentation